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ABSTRACT 



This paper synthesizes research on how motivation influenced 
teachers at two school -based performance award (SBPA) programs in Kentucky 
and in North Carolina. The research was conducted between 1995 and 1998 by 
the Consortium for Policy Research in Education. SBPA programs provide 
teachers and other school staff with pay bonuses for the achievement of 
specific schoolwide educational goals. The paper focuses on teachers' 
reactions to SBPA programs and the degree to which teachers ' motivational 
reactions are related to subsequent school performance. A combination of 
onsite interviews and survey questionnaires were used to assess teacher and 
principal attitudes and responses to existing programs and to determine 
whether the programs motivated teachers to work toward improved student 
achievement. The article provides a description of the programs in Kentucky 
and North Carolina, exploring whether those schools with more motivated 
teachers were more successful in improving student achievement. The report 
also examines whether teachers' motivational responses were predictive of 
schools* success in meeting their student achievement goals. The findings 
suggest that successful SBPA programs are complex undertakings that require 
attention to all five elements of the motivation model- -expectancy, 
instrumentality, valence, teacher competencies, and enabling conditions- -for 
it to work. (Contains 55 references, 13 tables and 1 figure.) (RJM) 
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SCHOOL-BASED PERFORMANCE AWARDS: 
RESEARCH FINDINGS AND FUTURE DIRECTIONS 



INTRODUCTION 

Building on a strong foundation of research and experience, in the 1990s educational 
policymakers undertook sophisticated comprehensive educational reform efforts. The centerpiece of 
these reforms was the creation of state educational standards and assessments, and the focused use of state 
and local resources to support the achievement of these standards. As part of these reforms, a number of 
states and districts have embraced the idea of performance-based accountability (Fuhrman, 1999), in 
which rewards (and sanctions) are used to motivate measurable improvements in student performance. 
Learning from the failure of efforts in the 1980s to link pay to performance at the individual teacher level, 
states and districts now are introducing policies that link pay to performance at the school level. This 
paper synthesizes research on the motivational effects on teachers of two such school-based performance 
award (SBPA) programs, in Kentucky and Charlotte-Mecklenburg, North Carolina. The research was 
conducted by the Consortium for Policy Research in Education between 1995 and 1998. 

What are School-Based Performance Award Programs? 

School-based performance award (SBPA) programs provide teachers and often other school staff 
with pay bonuses for the achievement of specific school-wide educational goals. These programs differ 
from individual performance pay in that all teachers in a school receive the bonus based on meeting an 
objective, predefined school goal, rather than individual teachers receiving a bonus based on individual 
teacher performance, as in merit pay programs. 

SBPA programs are meant to provide an incentive for teachers to work toward improved student 
performance, while avoiding the divisiveness that can accompany individual performance pay. The group 
nature of the goal is designed to encourage teachers to work together and to provide an important 
symbolic focus on key educational outcomes (Odden & Kelley, 1997). The interest in SBPA programs in 
the education sector parallels the rising use of group performance awards in the private sector. In that 



ERIC 



1 



■ 0 

xJ 



sector, the reported results of group rewards has generally been positive (Heneman, et.. al, in press, 
McAdams and Hawk, 1994). The research reported here is an assessment of the effect of these programs 
in public schools. 

Specifically, this paper focuses on the motivational reactions of teachers to SBPA programs and 
the degree to which teachers’ motivational reactions are related to subsequent school performance as 
defined by the program. The research does not address the question of whether SBPA programs “cause” 
improvements in student achievement. To do so would require the use of an experimental or quasi- 
experimental research design, which was not possible given the absence of equivalent or nearly 
equivalent states or districts to serve as comparisons. Instead, we used a combination of on-site 
interviews and survey questionnaires to assess teacher and principal attitudes and responses to existing 
programs, and to determine whether the programs created conditions likely to motivate teachers to work 
toward improved student achievement. We also aggregated these responses to the school level in order 
to examine whether, in Kentucky and Charlotte-Mecklenburg, those schools in which teachers were more 
motivated, on the average, were more successful in improving student achievement. In this article we 
provide a description of these programs and a summary synthesis of the methodology and findings from 
our studies of teachers’ motivational responses to the programs (Heneman, 1998; Heneman & 
Milanowski, 1998; 1999; Kelley, 1996; 1998a; 1998b; Kelley, Conley &. Kimball, 1999; Kelley & 
Protsik, 1997). We also provide a report for the first time about how teachers’ motivational responses 
were predictive of schools’ success in meeting their student achievement goals. 

The next section provides a brief overview of the Kentucky and Charlotte-Mecklenburg SBPA 
programs. Both programs have undergone modifications since 1998; the summaries reflect program 
design during the research period. 

The Kentucky SBPA Program 

Created in 1990 as a part of a court-ordered overhaul of the education system, the Kentucky 
accountability program provided rewards to schools for improved student performance, and sanctioned 
schools for declining performance. Performance was measured using an index developed from student 
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assessment scores in seven academic subjects (reading, writing, math, science, social studies, 
arts/humanities, and vocational/practical living) and school-level indicators such as student attendance, 
retention, dropout rates, and successful transition to adult life (see Kelley, 1998, Kelley and Protsik, 

1997). The school performance index, the Kentucky Instructional Results Information System (KIRIS), 
was administered in grades 4-5, 7-8, and 1 1-12, and consisted of portfolio entries, open-response, and 
multiple choice questions. Students scored in one of four categories on the assessment: novice, 
apprentice, proficient, and distinguished. The academic portion of the KIRIS index was calculated based 
on the proportion of the school’s students that scored in each category multiplied by a weight that gave 
more credit to performance at the higher levels. 

Beginning in 1991-92, the state set a series of 2-year goals for each school to increase its KIRIS 
score 10% of the distance between the school’s baseline score (initially the 1991-92 score, reset every two 
years) and a long-term target (equivalent to 100% of students scoring at the “proficient” level). If schools 
exceeded their goal, they were designated “reward” schools, and received a pool of reward funds that 
could be used for any purpose, including salary bonuses, according to a vote of the school’s certified staff. 
The reward amounts provided to each school were based on the degree to which the school exceeded its 
goal and the number of certified staff in the school. The maximum reward amount was 10% of the 
average teacher salary of the five Kentucky districts with the highest average salary ($3,690 for Cycle 1), 
although the program was never funded at this level. The minimum amount was 50% of this maximum. 
Certified staff at each reward school could decide how the award was to be used, including whether 
teachers only, or teachers and support staff, would be paid bonuses. In the first accountability cycle, 39% 
of schools exceeded their goal and received performance awards, and teachers in over 98% of award 
schools voted to use all or part of the money for salary bonuses. In the second cycle, 40% of schools 
exceeded their goal, and 88% of the schools voted to use all or part of the money for salary bonuses 
(Kentucky Department of Education, 1996; Reidy, 1995, Personal communication with Brian Gong, 
Associate Commissioner of Education, Kentucky Department of Education, August 1, 1997). The 
average bonus paid to teachers at the end of Cycle 2 (1995-96) was approximately $2,600. 
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For schools that failed to exceed their goal, there were four non-award categories. Schools that 
met but did not exceed their goal were labeled “successful,” and received neither rewards nor sanctions. 
Schools that improved over their baseline but failed to reach the target, “improving” schools, had to 
submit a school improvement plan to the state, indicating how they planned to meet their achievement 
goals in the future. Schools that dropped below baseline were designated “in decline” and were assigned 
a distinguished educator, received improvement funds, and had to submit a school improvement plan. 

The distinguished educator was an exceptional educator from the state of Kentucky hired and trained by 
the state to provide technical assistance to schools to help them meet accountability goals. Schools that 
dropped more than five points below baseline were designated “in crisis.” They were treated like decline 
schools, with one notable exception. The distinguished educators assigned to crisis schools had broad 
powers to terminate teachers and override school site council decisions if they deemed it appropriate 
(Kentucky Department of Education, 1995). In the first biennium of the program, the crisis category was 
suspended due to concerns about the reliability of the assessment instrument. The crisis category was 
reinstituted for the second biennium; 177 were designated in decline and nine schools were designated in 
crisis (Kentucky Department of Education, 1995; 1996; Lindle, 1999). By the end of the cycle, 88% of 
these schools had improved enough to receive rewards, suggesting that “the DE program would appear to 
be a smashing success” (Lindle, 1999, p. 11). 

Our research, conducted between 1995 and 1998, examined the program as described above. In 
1998, a variety of factors led to a restructuring of the assessment and accountability program (Blackford, 
1997, Lawton, 1997, Strecklow, 1997). The new program includes a new assessment with stronger links 
to nationally norm referenced tests, and changes in the nature of rewards and sanctions, including the 
elimination of pay bonuses for teachers in favor of rewards paid directly to schools (Cody, 1998; Lindle, 
1999). 

The Charlotte-Mecklenburg SBPA Program 

Under the direction of a new superintendent, the Charlotte-Mecklenburg School District (CMS) 
established its Benchmark Goals Program (BGP) in 1991 as a way of attempting to reverse a record of 
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low student achievement and limited success with minority students (Heneman, 1998). The BGP became 
operational in 1992-93, with annual plans for implementation and management extending through the 
1996-97 school year. In 1996-97, the BGP entered a transition period due to the hiring of a new 
superintendent and the beginning of a new statewide accountability program for school districts called the 
ABC’s program, a quasi-acronym for accountability, emphasis on basics and high standards, and 
maximum local control. This program changed many of the aspects of the BGP for the 1997-98 school 
year, but the major elements of the original BGP, described below, remained intact during the period of 
our study. 

In the program, improvement goals for student achievement were set in nine areas: reading, 
writing, math, social studies, primary grade readiness, higher level course enrollment, end of course 
subject mastery, attendance, and dropout rates. Depending on grade level composition, schools had 
between 14 and 44 subgoals. Student achievement in cognitive areas was assessed by the state 
standardized multiple-choice tests. Initial performance baselines were established for each school in 
1991-92, with annual improvement goals reset each year by the district. Maintenance goals were 
established in instances in which the school was already performing at a high level. There were separate 
improvement goals for white, African-American, and other students; maintenance goals were common for 
all students. 

Schools received points for achieving each subgoal to a maximum of 100 points (e.g., if a school 
had 25 subgoals, each subgoal was worth 4 points). A fully met subgoal received 100% of the points, a 
partially met goal received 75% of the points, and an unmet goal received no points. Points earned on 
subgoals were summed to form a school’s total score. Schools that earned 75 to 100 points were 
designated as “exemplary;” schools that earned 60 to 74 points were designated as “outstanding.” 

Schools that earned fewer than 60 points were in the no bonus category (there was no special 
designation). Teachers in exemplary schools received $1,000 and support staff received $400. Teachers 
in outstanding schools received $750 and support staff received $300. 
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There were no formal sanctions for schools that failed to achieve the accountability goals for any 
one time period. However, schools with chronically low achievement could be placed in the district’s 
Priority Schools program, and in 1995-96, 14 schools were so designated. Guided by assistance from the 
district, these schools prepared school improvement plans, identified resources needed to improve school 
performance, and received staff development from a team of university researchers. 

Motivation Model 

Qualitative research on the motivational effects of SBPA programs on teachers in Kentucky 
(Kelley & Protsik, 1997) suggested that expectancy theory (Vroom, 1964) and goal setting theory (Locke 
& Latham, 1990) could be useful in understanding teacher motivational response to SBPA programs. 
Figure 1 shows the resulting model of motivation we used to structure further research. The model 
depicts factors involved in teacher motivation to work toward student achievement goals. As previously 
described (Heneman, 1998), the central elements of the model are teachers’ perceptions about the degree 
to which their own efforts will produce student performance as defined by the SBPA program (the 
expectancy perception), teachers’ perceptions of the probability that meeting the student achievement 
goal will lead to desirable outcomes such as receiving a bonus (the instrumentality perception), and the 
desirability that teachers attribute to these outcomes (the valence perception). 

Insert Figure 1 about here 

Thus, in this model, teachers’ motivation is a function of expectancy, instrumentality and valence. 
Starting with expectancy (the effort to achievement link), the teacher’s decision to expend effort is 
determined by beliefs about the likelihood that the teacher’s effort (its intensity, focus and persistence) 
will lead to the attainment of student achievement goals. The more a teacher perceives that working hard 
will pay off in meeting the goals, the greater the teacher’s motivation to expend the effort needed to do so. 
Moreover, borrowing from goal-setting theory (Locke & Latham, 1990), student achievement goals that 
are clear and specific will focus teachers’ efforts and foster stronger expectancy perceptions. 

Teachers’ beliefs about the effort-student achievement link are influenced by their beliefs about 
their own competencies and the presence of enabling conditions. Competencies refer to teacher 
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knowledge and skills relevant to helping students meet the achievement goals. Enabling conditions are 
the organizational conditions that facilitate the meeting of student achievement goals. Examples include 
principal support, professional development, and feedback about progress toward meeting the goals. 

The instrumentality link is also important in understanding the motivational impacts of SBPA 
programs. Instrumentality is the perception that meeting (or not meeting) student achievement goals will 
lead to the outcomes associated with the program (e.g. receiving a pay bonus, experiencing public 
criticism for failing to meet improvement goals). To the extent that teachers believe that consequences 
they value have a high probability of occurrence as a result of meeting the goals, they will be motivated to 
expend effort. Similarly, to the extent that they believe that failure to achieve the goals will result in 
consequences that they disvalue, they will be motivated to try to achieve the goals. The model thus 
suggests that outcomes will motivate to the extent that teachers see them as highly likely to occur as a 
result of achieving (or failing to achieve) the goals. 

Finally, teachers must value the outcomes or consequences. Examples of positively valued 
(desirable) outcomes of SBPA programs might include receiving a bonus, public recognition for 
achieving student achievement goals, and a sense of accomplishment from having students reach the 
goals. Negatively valued (undesirable) outcomes might include public criticism if student achievement 
goals are not met, and threats to job security. Individual teachers are likely to vary in their assessment of 
the desirability of any one outcome. Some outcomes may even be viewed as desirable to some and 
undesirable to others (e.g., a degree of stress may be viewed as negative to some teachers, and as 
producing a positive, stimulating work environment to others). 

The motivation model has important implications for the design of, and context for implementing 
SBPA programs. Policymakers need to ensure that the outcomes associated with reaching accountability 
goals are highly valued by teachers, and are seen as highly likely to occur if the goals are reached. They 
must also try to minimize likely negative consequences of trying to reach the goals (such as higher stress), 
or ensure that the positive outcomes “outweigh” the negative outcomes. And attention must be paid to 
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teacher competencies and to providing enabling conditions that foster teachers’ beliefs that their efforts 
will lead to goal achievement. 

Using this conceptual model, our research sought to address two connected issues. First, what are 
the motivational responses of teachers to an SBPA program? And second, are teachers’ motivational 
responses predictive of their schools’ subsequent success in meeting student achievement goals? 

RESEARCH DESIGN AND METHODOLOGY 
Teachers’ Motivational Responses 

Mixed qualitative/quantitative research methods were used to study the motivational effects of 
SBPA programs in Kentucky and CMS. The research design and methods are summarized here; a more 
detailed description can be found in Kelley, Heneman & Milanowski (1999). 

The research was conducted between the beginning and the end of a single program cycle in the 
two jurisdictions. This allowed us to frame our interviews and survey questions in terms of that 
performance cycle, and to link perceptual data to student achievement outcomes at the end of the cycle. 
We began data collection by conducting interviews with teachers and principals in schools that varied by 
student socioeconomic status, school level, grade level, and previous school success in meeting student 
achievement goals. Interviews were conducted in 16 schools in Kentucky in 1996 (see Kelley, 1998) and 
12 schools in CMS in 1997 (see Heneman, 1998). Researchers spent a half-day in each school, 
conducting 45 to 60 minute interviews with the principal and a sample of teachers, particularly those who 
taught in the accountability subjects/grades. Interviews were taped, transcribed, and analyzed to provide 
input into the development of the survey instruments. 

Separate survey instruments were developed for teachers and principals in Kentucky and CMS. 
Where possible, scale items were adapted from preexisting survey instruments. Common questions were 
included across instruments, although the questions were tailored to be descriptive of the programs 
operating in each research site. The survey included questions designed to measure the various constructs 
of the motivation model (perceptions of goals, expectancy, instrumentality, valence, teacher 
competencies, and enabling conditions). 
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The surveys were administered to CMS teachers in Spring, 1997 to 25 teachers selected at 
random from all 120 schools. If a school had 25 or fewer teachers, all of the teachers in the school were 
selected. 1 150 teachers responded to the questionnaire for a 39 percent response rate. The respondents 
matched the CMS teacher population relatively well by gender, race, and teaching service, but elementary 
teachers were somewhat over-represented and high school teachers under-represented. 

The questionnaires were administered to Kentucky teachers in Fall, 1997. Respondents were 
selected by means of a multi-stage sampling process designed to provide representation from each 
performance category and grade level (see Kelley, Heneman & Milanowski, 1999). As in CMS, 25 
teachers in each school were selected, or for smaller schools, all of the teachers in the school. The total 
number of useable responses was 1,750, for a response rate, relative to the number of questionnaires sent, 
of 30.9%. At least one response was received from 261 of the 262 schools sampled. In terms of 
demographics, respondents were a good match to the population of teachers in Kentucky. 

Motivation and Student Achievement 

In addition to examining the effects of SBPA programs on individual teacher motivation, we also 
explored the question of whether teachers’ motivation affects subsequent school performance, as defined 
by the SBPA program. To investigate this, teacher motivational responses were aggregated to the school 
level, then used to predict subsequent school performance at the end of that accountability cycle. The 
global hypothesis drawn from our motivational model was that the greater the average teacher motivation, 
the greater the school’s success in improving student achievement levels. 

The motivational constructs for which school-level averages were calculated included teachers’ 
individual expectancy perceptions and the expected value of outcomes (valence times instrumentality). 
The expected value of the bonus alone was calculated, along with an expected value for each of four 
outcome factors associated with achieving, striving to achieve, or failing to achieve program goals. The 
expected value of each outcome in the group was calculated by multiplying the desirability rating times 
probability of occurrence, then summing the products within each factor grouping. Then, the individual 
scores were averaged across teachers within each school. 
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Whenever individual-level data are aggregated to form a score representing a higher-level entity 
such as a school, the question of the degree of agreement in the responses of the individuals arises. Here 
we are using the within-school averages of teachers’ ratings of their motivational responses to represent 
the average level of motivation within each school. Within-school variability in individual teachers’ 
responses leads to sampling error in the estimate of each school-level average. This is a form of 
unreliability that is likely to attenuate the relationships between the school-level averages of the 
motivational constructs and the measures of school performance. 

School performance was defined by the SBPA program rules. For CMS, school performance was 
measured as the number of Benchmark Goals points each school achieved in the 1996-97 school year, as 
reported by the District. For Kentucky, the indicator was the change in the KIRIS index between the 
baseline (essentially the KIRIS index from Cycle 2) and the index for Cycle 3 (consisting of the average 
of the 1997 and 1998 indices). 

In these analyses, we controlled for a number of school characteristics that might be expected to 
affect how well schools did on the accountability indices. These included school size, operationalized as 
the natural log of enrollment, and the percentage of students receiving free and reduced price lunch. For 
the Kentucky analysis, the value of the Cycle 2 KIRIS index was also included, because it may have been 
easier for schools with relatively low levels on the Cycle 2 index to improve enough to be a reward school 
than it was for schools with higher levels on the index. In CMS, many of the improvement goals 
underlying the Benchmark Goals points are adjusted by the district to account for this possibility, so 
previous school performance was not used as a control variable. 



FINDINGS: TEACHERS’ MOTIVATIONAL RESPONSES 
Teacher Perceptions of School Performance Goals 

Our interviews with teachers and principals suggested that the Kentucky and CMS SBPA 
programs were reasonably successful in getting teachers to understand and be committed to achieving the 
goals. All of the teachers we interviewed in both jurisdictions were aware of and had a significant degree 
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of understanding of program goals. In both Kentucky and CMS, many teachers indicated that the 
presence of the SBPA program provided them with clear direction and focus linked not just to covering 
the material, but to reaching measurable student outcomes. Many principals and teachers named 
accountability and goal focus as major positive effects of the SBPA programs. While teacher 
commitment to these goals varied across schools, grade levels, and subjects, many of the teachers we 
talked to indicated that they had made changes in teaching practice to align curriculum and instmction 
with program goals. In the most successful schools that we studied, changes to teaching practice occurred 
school-wide. Despite these substantial impacts, a degree of perceived goal conflict was expressed by 
some of the teachers. Among the competing goals mentioned were the need to emphasize basic skills, the 
importance of instilling a “love of learning”, or pursuing a different program emphasis, such as in magnet 
schools (Heneman, 1998, Kelley, 1998). 

The survey results were mostly consistent with the interview findings. Teachers in both 
jurisdictions were asked about their comnutment to achieving the goals, how well they understood the 
goals, and whether these goals conflicted with other important educational goals. Table 1 shows the 
means and standard deviations on the scales we developed from these items. (See Appendix A1 for 
information on the items included in the scales and their reliabilities.) 

Insert Table 1 about here 

As can be seen, on average, teachers reported a high level of comnutment and a reasonably high 
level of understanding of the goals. Also, teachers in both jurisdictions reported a modest degree of 
conflict between the accountability goals and other important educational goals. Nonetheless, reported 
levels of goal comnutment and clarity are encouraging in that policy implementation research has 
repeatedly underscored how difficult it is for policy to penetrate teaching practice (Fullan, 1991; Odden, 
1991). 

Teachers in both jurisdictions were also asked about the level of performance they were striving 
for during the accountability cycle. Table 2 shows the percentages of respondents who indicated they 
were trying to attain the different levels of achievement established by each program. Inspection of the 



table reveals that in both jurisdictions, the vast majority of teachers indicated they were trying to reach 
their school’s student achievement goals. Over 87% were trying for “Outstanding” or “Exemplary” status 
in CMS, and over 77% for “Successful” or “Reward” status in Kentucky. Few indicated that they were 
not trying or were trying only to avoid performance declines. Moreover, goal attainment experience in 
the last accountability cycle had a strong effect on performance targets. Teachers in CMS were more 
likely to report trying for “Exemplary” status if their school had earned (79.6%) versus not earned 
(68.4%) a bonus in the previous year. The effect of prior winning was even greater among Kentucky 
teachers, with “Reward” status sought by many more teachers in schools that had won (72.2%) than had 
not won (41.4%) a bonus for the previous cycle. 

Insert Table 2 about here 

The use of bonuses may help to increase commitment to goals and teacher’s understanding of 
them by reinforcing their presence and signaling their importance. To investigate this possibility, we 
multiplied bonus valence times instrumentality to create an expected value of the bonus. Goal 
commitment and goal clarity were regressed on this expected value and on several control variables. 
Significant positive coefficients were obtained for the effect of the expected value of the bonus on goal 
commitment in both jurisdictions, and for goal clarity in the CMS sample. (The goal clarity coefficient in 
the Kentucky analysis just missed significance at the .10 level.) These results suggest that bonuses can 
play a positive role in enhancing teacher understanding and commitment to the goals. 

A potential threat to goal commitment over time is the difficulty of meeting ever-rising goals for 
successful schools. In both SBPA programs, for schools that do improve, the absolute level of student 
achievement that must be produced to meet future goals will be higher in each succeeding accountability 
cycle. For schools that do not improve in a cycle, the goal for the next cycle does not increase much, if at 
all, in terms of absolute student performance. To assess the impact of program characteristics on 
perceived goal difficulty, teachers were asked whether they thought the goals were harder this cycle than 
last, and would be harder yet next cycle. Table 3 shows the percentages of teachers who indicated that 
future goals would be more difficult, by the reward status of their school in the last accountability cycle. 
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It can be seen that in both jurisdictions, teachers in rewtu'd schools reported greater goal difficulty than 
teachers in nonrewtu'd schools. 

Insert Table 3 about here 

Teacher Perceptions of Outcomes: Valence and Instrumentality 

Valence . Our analysis of teacher interview responses indicated that teachers associated a variety 
of outcomes with the programs (see Heneman, 1998; Kelley, 1998; Kelley & Protsik, 1997). Based on 
these findings, 17 potential outcomes directly relevant to experiences under the SBPA programs were 
identified and included in the questionnaires sent to Kentucky and CMS teachers. Teachers rated the 
desirability of each outcome on a 1-5 scale. To determine if there were meaningful clusters of outcomes, 
the ratings were factor analyzed (principal components, varimax rotation). All outcomes except one 
(“participation in meaningful education reform”) had a high (> .50) loading on only a single factor and 
were retained. The factors were labeled Goal Attainment Rewtuds, Learning, Sanctions, and Stress. 
Means and standtu*d deviations for the 16 outcomes comprising these four factors are shown in Table 4 
(see Heneman & Milanowski, 1999). 

Insert Table 4 about here 

It can be seen that Goal Attainment Rewtuds include both extrinsic (bonus, school improvement 
funds, public recognition) and intrinsic outcomes (satisfaction from meeting goals and from improved 
student performance). These results tu-e consistent with our interview findings. The Learning outcomes 
predominantly intrinsic in nature, focusing on both learning for the students and for the teachers. The 
Sanctions outcomes were rated quite low in desirability, particultu'ly in Kentucky where the sanctions in 
reality were more powerful than they were in CMS. The Stress outcomes were uniformly rated very low 
in desirability, a finding that ptu*allels the interview findings. 

Finally, we should note that outcome desirability ratings were not strongly related to teachers’ 
age, yetu's of experience, salary level, school level, percent of students receiving free and reduced price 
lunch, and whether a school had met its goals in the previous award cycle. Very few statistically 
significant correlations between desirability ratings and these variables were obtained, and of those that 
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were significant their size was very small (i.e., r < .15). Thus, these desirability ratings appear to be quite 
generalizable across teachers within each jurisdiction. 

In the interviews, teachers were asked about the likelihood of various outcomes if their school 
met, or did not meet its student achievement goals. We also explored the likelihood of outcomes 
occurring as a consequence of trying to reach the student achievement goals. On the survey, an 
instrumentality perception was gathered for each of the 17 outcomes. Depending on the outcome, 
teachers were asked to indicate the probability of occurrence if the school met its goals, did not meet its 
goals, or the likelihood of occurrence in the process of trying to meet school goals. The probabilities, 
shown in Table 5, were assessed on a 0% to 100% scale, in 10% increments. 

Insert Table 5 about here 

The probabilities of the Goal Attainment Rewards occurring if a school met its goals were 
perceived to be very similar in Kentucky and CMS, except for the bonus, which was much higher (72.8%) 
in CMS than in Kentucky (54.8%). This may be due in part to uncertainty surrounding the future of the 
Kentucky SBPA program at the time of our survey. The perceived probabilities of the Learning outcomes 
were also similar for Kentucky and CMS teachers. For the Stress outcomes, Kentucky teachers reported 
higher probabilities of increased pressure and stress and working more hours. Perceived probabilities of 
outcomes associated with not meeting goals (i.e.. Sanctions outcomes) were substantially higher for 
Kentucky teachers, probably due to the formal sanctions components of the Kentucky program. 

Correlations between probability perceptions and teacher age, years of experience, salary, school 
level, percent free and reduced lunch, and previous reward status were examined. Most of the 
correlations were not statistically significant, and of those that were significant their size was quite small 
(i.e., r < .15). The one exception was a moderate and significant correlation (r = .23 for CMS teachers, r 
= .30 for Kentucky teachers, p < .05) between having earned a bonus in the previous cycle and the 
probability of receiving a bonus for meeting school goals in the current cycle. 

The interview and survey findings converged and complemented one another. The most striking 
finding in both program sites was the relatively low perceived probability of receiving a bonus when 



O 

ERIC 



14 



16 



school goals are met. Moreover, there was only a moderate correlation between perceived probability of 
receiving the bonus for achieving the goals and having received a bonus in the past award cycle. These 
data suggest teacher skepticism regarding the bonus portion of the program. Even reaching goals and 
winning bonuses last time was not a very strong influence on believing that history would repeat itself. A 
second striking Ending was that Kentucky teachers, relative to CMS teachers, perceived much greater 
probabilities of occurrence of negative outcomes for trying to meet, and for not meeting program goals. 
Specifically, more work hours and more pressure and stress were seen as more likely than any of the 
positive outcomes. This may have been due to differences in the difficulty of achieving goal targets 
between the two programs, and to the more highly politicized context of the Kentucky program. 

Kentucky teachers also saw a higher probability of sanctions for poor performance than CMS teachers, 
reflecting the stronger sanctions provided by the Kentucky program. 

In summary, we clearly and convincingly found that an SBPA program is not just a bonus 
program; rather it is one element in an interrelated system of rewards, opportunities, and demands that 
influences teachers’ jobs and lives in a multitude of ways, leading them to experience (and form values 
about) a variety of both extrinsic and intrinsic outcomes. We also found that most teachers viewed 
receiving a bonus as appropriate and desirable, and not in conflict with other outcomes traditionally seen 
as motivating to teachers. However, we were surprised to find, as discussed later, low levels of 
desirability and motivation being reported for the bonus program overall. This seeming inconsistency 
was found to be a reflection of less than favorable attitudes about components of the SBPA program, and 
how it is administered, rather than of the bonus per se. 

The Motivational Role of the Bonus 

Given the central role of bonuses in SBPA programs, and the lack of evidence available to date 
about their motivational impact on teachers, we explored teachers’ reactions to the bonuses in depth. Part 
of this exploration occurred during our interviews with teachers in the two programs. From these 
interviews, we learned that: 
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• Teachers felt it was appropriate to receive bonuses, and that receiving a bonus was deserved. 

• The teachers varied in how meaningful the size of the bonus they could receive actually was, 
especially after deductions for taxes. 

• The teachers varied in how much the possibility of earning a bonus motivated them to improve 
student achievement, or whether an even larger bonus would motivate them more. 

• Teachers were skeptical that earned bonuses would actually be paid, due to past experiences with 
actual reneging on bonus payments or beliefs that the funding for the bonuses would not be 
continued. 

• Teachers varied as to whether they wanted the bonus part of the SBPA program to continue. 

• The meaning of the bonus varied, with teachers variously viewing it as an appropriate “thank you”, a 
formal recognition, reimbursement for personal expenditures on school-related items, a reward that 
allowed for the purchase of desired goods, or that it was simply irrelevant. 

• Teachers in Kentucky found that having to decide among themselves how to divide up the bonus 
money among teachers and staff was a divisive process that created tension within and between 
schools (see also Abelmann & Kenyon, 1996). 

• Awards paid as salary bonuses appeared to have more visibility than awards paid as school 
improvement funds. Kelley, Conley, and Kimball (1999) found that Maryland teachers were much 
less cognizant of the award than Kentucky or CMS teachers. Unlike Kentucky and CMS, the 
Maryland program paid awards in the form of school improvement funds. 



Teacher Perceptions of Program Fairness 

The surveys also included a set of items dealing with the perceived fairness of various aspects of 
the SBPA programs. Table 6 shows the means and standard deviations of teacher responses to these 
scales; items used to form the scales and reliabilities are shown in Appendix A2. 

Insert Table 6 about here 

While CMS teachers on average expressed weak agreement that it is fair to hold teachers 
accountable for student performance, and to pay bonuses to teachers when student performance improves, 
Kentucky teachers on average tended to disagree with both ideas. Teachers in both jurisdictions were just 
below the neutral point with respect to whether they felt their school had a fair chance of winning the 
award and that the program was run fairly. Additional analysis revealed that teachers in schools that had 
consistently won awards had slightly more positive perceptions of fairness than teachers in schools that 
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had not won awards, although the averages for the program fairness scale were still below the neutral 
point for both jurisdictions. This suggests that many teachers had some serious reservations about the 
fairness of these programs. 

To delve more deeply into teachers’ reactions to the program and to the bonus, two additional sets 
of analysis were conducted. The first sought to predict teachers’ motivational response to the program, 
whether they felt the bonus part of the program should continue, and their probability of withdrawing 
from the program. The independent variables pertained primarily to various elements of perceived 
fairness of the program. The analysis, described in more detail in Heneman and Milanowski (1999), was 
conducted among a subset of Kentucky teachers. 

On average the teachers did not report being highly motivated by the bonus program, but were 
more neutral in their intentions to withdraw from the program and their desire to see it continue. It should 
be remembered that these data were collected at a time of contention and uproar about the KIRIS program 
(Blackford, 1997, Lawton, 1997, Strecklow, 1997), and it is unknown whether more favorable attitudes 
would have been expressed had the data been collected earlier. There was considerable variability in 
teachers’ overall attitudes about the bonus program, however, and several factors accounted for a 
significant portion of this variability. Teachers were both more motivated by the program and wanted it 
to continue if they were dissatisfied with their current salary, felt that use of bonuses and the bonus 
process was fair, and felt that teachers should decide how to divide up the bonus money. In contrast, 
teachers’ withdrawal intentions were related to perceptions that the bonus process was unfair, that their 
school did not have a fair chance to meet its goals and the existence of unfair winners (Heneman & 
Milanowski, 1999). 

A second set of analyses, conducted among CMS teachers, examined longitudinally the factors 
causing job withdrawal (see Heneman & Milanowski, 1998). The independent variables were salary, 
satisfaction with salary, whether the teacher was working in a bonus-winning school, perceptions of the 
fairness of the bonus, and intention to leave because of the SBPA program. 
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The analysis of actual teacher withdrawal response in CMS showed several significant predictors 
of withdrawal. Teacher salary was significantly and negatively related to total turnover (leaving the 
district and transferring schools), and the negative pay coefficients for separate analyses of leaving the 
district and transferring schools just missed being significant. Non-bonus schools had significantly more 
total turnover, due to significantly greater teacher transfers out of those schools. Intent to leave the job 
within the next two years because of the award program had the strongest impact of all the variables on 
both leaving the district and on transferring schools. In short, both teachers’ salaries and their experiences 
with the SBPA program, including whether or not they were in bonus-winning schools, influenced their 
actual withdrawal behavior. 

Teacher Expectancy Perceptions 

The interview data in Kentucky and CMS suggested that teacher expectancy varied across 
teachers and schools. Schools that had achieved award status in the past were likely to have higher 
teacher expectancy than that had failed to improve. The Kentucky data were also analyzed to examine 
differences across schools with various award histories. These data suggested that award schools, in 
which teacher had relatively high levels of teacher expectancy, were characterized by principal and 
district support for the goals of the program, high quality professional development linked with the 
accountability program, meaningful collaboration among teachers, and connections to external 
professional communities. Non-award schools, in which teachers had relatively low levels of teacher 
expectancy, were characterized by an absence of the conditions described above, and multiple or 
conflicting goals that competed with the goals of the accountability program (Kelley, 1998). 

Teacher surveys administered in Kentucky and CMS were designed to further illuminate 
important organizational conditions related to teacher expectancy. Two measures of expectancy were 
examined: individual teacher expectancy and teacher group expectancy. Individual teacher expectancy is 
a measure of the extent to which a teacher believes that her efforts will result in achievement of school 
accountability goals. Teacher group expectancy is a measure of the extent to which an individual teacher 
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believes that she and her colleagues working together can achieve the school accountability goals. Table 
7 shows the average expectancy scores for Kentucky and CMS teachers. 

Insert Table 7 about here 

Expectancy levels were not particularly high in either jurisdiction, though the averages were 
considerably lower in Kentucky. In Kentucky, teachers on average did not report even a 50% probability 
that individual effort would lead to performance sufficient to achieve reward status, or even avoid being 
classified as a school in decline. They were, however, more positive about the likelihood that if they and 
their colleagues worked together, the school would achieve reward status or avoid sanctions. 

The level of expectancy varied greatly across teachers. According to our motivational model, 
potential determinants of teacher expectancy include both teachers’ assessments of their own skills and 
capacities and their assessments of whether important enabling conditions are present. Using the survey 
data, we explored the degree to which these factors influenced teacher expectancy in Kentucky and CMS 
(see also Kelley, 1999). Four types of independent v^ables were included in the analysis; school 
demographics, organizational context, teacher attitudes, and teacher characteristics. In both Kentucky 
and CMS models, demographic variables were included to control for the effect of differences in school 
level and prior experience with the reward program (e.g., prior achievement of award status), school size 
and the percent of students receiving free or reduced-price lunch. 

Organizational context variables included feedback, alignment, professional community, teacher 
participation in decision making, principal support, and goal conflict. All context variables represent 
equivalent constructs in Kentucky and CMS, except that additional professional community items were 
added to the Kentucky survey, and are included in the professional community variable for Kentucky in 
Model 2. Teacher attitude variables included teacher perceptions of program fairness and goal clarity. 
Teacher skill variables include years of teaching experience and highest level of education completed. A 
description of the items included in the various scales used in the analysis is found in Appendix A3. 

Table 8 contains the results of the regression analyses for individual expectancy in Kentucky and 
CMS. Regression analyses of group expectancy were also run; findings are similar to the individual 
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expectancy variable, and are therefore not reported here (see Kelley, 1999). Variables found to have a 
significant effect on expectancy in both jurisdictions include feedback, goal conflict, school management, 
leadership and culture, and the perceived program fairness. 

Insert Table 8 about here 

Feedback had the strongest effect on expectancy. Feedback involves the ability to receive and 
interpret results of past assessments, and use this information to refine curriculum and instructional 
programs. Goal conflict had a consistently significant dampening effect on individual teacher and teacher 
group expectancies. Goal conflict is the presence of organizational goals that compete with the goals of 
the accountability program. Competing goals can have a variety of causes. From the qualitative research 
we found schools with academic programs which ranged from very weak to very strong with a different 
emphasis from the accountability program. For example, teachers in magnet schools indicated that the 
presence of a strong magnet emphasis limited their ability to focus on the goals of the accountability 
program. 

Principal support had a positive effect on expectancies in CMS, and professional community had 
a positive effect in Kentucky. Professional community may have a stronger impact in Kentucky because 
the accountability program itself had a slightly different focus. The CMS program required less change in 
teaching practice, and so professional support was less important to school success. Among the teacher 
attitude variables, program fairness stood out as a strong predictor of expectancy. Fairness reflects the 
perception that the program is administered fairly, and that each school has an equal chance of doing well 
in the program. Interestingly, measures of teacher knowledge and skills, goal clarity, alignment, and 
student socioeconomic status (SES) were not found to be significant predictors of teacher expectancies. 
Findings: Teacher Motivational Response and School Performance 

School performance was regressed on the control variables, expectancy, and the expected value 
of outcomes. The results are shown in Table 9. (Means, standard deviations, and intercorrelations of the 
variables are shown in Appendix A4.) Along with a controls-only model, six regression models were 
estimated to reflect the six different treatments of expected value of outcomes (bonus only, goal 
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attainment only, learning outcomes only, sanctions only, stress only, and all four factors together). These 
separate analyses allowed us to probe for potentially different effects of each type of outcome. Only 
additive model results are shown, since the multiplicative terms (expectancy times expected value) did 
not significantly increase variance explained. 

Insert Table 9 about here 

The controls-only model accounted for significant variance in school performance, but adding the 
variables representing teacher motivation significantly increased the proportion of variance explained. In 
all models, average expectancy had a positive coefficient of substantial magnitude, and was significant at 
the .01 level. Thus, the higher the average teacher expectancy, the more Benchmark Goals points a 
school achieved (CMS) or the greater the increase in the KIRIS index (Kentucky). However, the 
expected value of outcomes variables had relatively small coefficients, and some had unexpected signs. 
We expected that schools in which teachers had on the average higher expected values for the bonus, goal 
attainment, and learning outcomes would perform better, controlling for expectancy. These expectations 
were not bom out. All of these outcome expected values had either insignificant, or significant but 
negative coefficients. 

With respect to the “sanction” outcomes, again the coefficients were small and not statistically 
significant. This result does not support the expectation that schools in which teachers, on the average, 
perceive these outcomes as having greater expected “disvalue” perform better. Finally, with respect to 
the stress outcomes, the coefficients were again small, and only statistically significant for the CMS 
schools. The negative signs are consistent with the expectation that schools in which teachers disvalue 
these experiences and see them as a likely consequence of trying to meet the goals are likely not to 
perform as well. Similar analyses done using only elementary schools (n=81 for CMS, n=83 for 
Kentucky) did not show any substantially different results from those reported above. 

Overall, these analyses strongly suggest that between-school differences in expectancy are related 
to the success of schools in improving student performance as defined by the school-based performance 
award programs in each of these sites. However, between-school differences in the degree to which 
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teachers value and expect to receive the various outcomes, including bonuses, associated with the goal 
achievement appear to be mostly unrelated to differences in school goal achievement. These latter results 
are contrary to our theoretical expectations. 

STRENGTHS AND LIMITATIONS OF STUDY METHODS 

These results should be interpreted within the context of the strengths and limitations of the study 
methodology, and of the specific designs of the SBPA programs studied. On the strength side, this study 
is characterized by construction of a teacher motivation model to serve as a guiding framework to study 
design and measurement, using a predictive design to assess the relationship between teacher motivation 
and student achievement, large samples from multiple research sites, a meshing of qualitative and 
quantitative data collection and analysis, and multivariate empirical analyses at both the individual and 
school level to fully explore the complex interrelationships embedded within our data. 

These strengths notwithstanding, certain limitations of our study should be borne in mind. A first 
limitation pertains to measurement. On the independent variable side, our measures for the most part were 
ad hoc in nature, a common characteristic of expectancy and goal setting theory studies. As a result, very 
little is known about their construct validity. Additionally, operationalizing expectancy theory requires the 
use of multiplicative combinations of variables, which in turn exacerbates unreliability of measurement. 
We suspect that such unreliability is the major cause of the weak relationship of expected value of 
outcome measures to school performance. Our finding that schools in which teachers value the rewards 
associated with the SBPA program do not appear to have higher levels of performance may be more a 
reflection of measurement error and research design than of a lack of a substantive relationship. 

Measurement concerns also surface in the case of our student achievement measures at the school 
level. We relied upon existing achievement measures used by the Kentucky and CMS SBPA programs. 
We had no reliability or validity data for the CMS measures. The information available for Kentucky’s 
KIRIS gain scores (Kentucky Department of Education, 1997) suggests that while they are fairly reliable, 
they are subject to a non-negligible degree of measurement error. In turn, this may have attenuated the 
predictive relationships between teacher motivation and subsequent school success. (See Milanowski, 
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1999 for a discussion of how the reported level of reliability affects the categorizations of schools as 
eligible for receiving a performance reward.) 

A second major limitation is our lack of control or comparison groups. The ideal situation would 
have been to study our three SBPA programs alongside identical programs without key features such as 
the bonus. By studying only sites that had SBPA programs, we had to rely on naturally occurring 
variation within these programs (such as differences in teacher expectancy, perceived probability of 
receiving a bonus after meeting goals, and outcome desirability) to infer how "motivational" the program 
was and to estimate the impact of teacher motivation on subsequent school success in meeting 
achievement goals. The availability of comparison or control groups would have allowed us to better 
pinpoint the trae magnitude of the SBPA's impact on teacher motivation and the impact of teacher 
motivation on school success. 

A third limitation pertains to the generalizability of our findings. One constraint on 
generalizability is the time-bound nature of our data. The data were gathered midstream during the 
operation of the SBPA programs, and it is uncertain whether our data and resultant relationships would 
have been the same at either earlier or later points in time. For example, the CMS data were gathered as 
the Benchmark Goal Program was undergoing a transition to integration with a statewide accountability 
plan. Now that the integration has been completed, we are uncertain as to how well our results generalize 
to this new circumstance. In the case of Kentucky, we gathered our data at the time of substantial 
controversy over the KIRIS testing process; had we gathered our data a year earlier, prior to the 
controversy, teachers’ responses might have been very different. Another constraint on generalizability is 
the nature of our samples and their accompanying demographic and SBPA program characteristics. 
Though we sampled carefully within our jurisdictions, thus enhancing generalizability to our jurisdiction 
populations, we have no way of knowing how well our results might generalize beyond the sites we 
studied to other samples and other SBPA programs. 
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CONCLUSIONS AND IMPLICATIONS 



Despite these limitations, our data suggest that SBPA programs are a potentially useful tool for 
policymakers and administrators who are interested in focusing teachers, principals, and educational 
systems on improving student performance. However, the full potential may be difficult to achieve, due 
to the complexity of designing and implementing SBPA programs effectively. We found that the 
programs we studied had mixed success in fulfilling the conditions that our motivational model suggests 
are important. Teachers reported that they understood program goals and were committed to them. The 
programs appear to have a focusing effect on teaching practice, collaboration and teamwork, and resource 
alignment. Although most teachers reported trying to reach the goals, many were skeptical about 
continuing to reach ever-rising goals. This skepticism was higher in schools that had previously met their 
goals. 

Teachers associated both desirable and undesirable outcomes with the program, and teachers 
indicated that they valued both intrinsic and extrinsic outcomes. The outcomes grouped into four clusters: 
Goal Attainment, Learning, Stress and Sanctions. Outcomes in the first two clusters received high 
desirability ratings and outcomes in the last two clusters received low desirability ratings. The most 
desirable outcomes included personal satisfaction, seeing students learn new skills, having clear goals, 
and receiving a bonus. Least desirable outcomes included public criticism or embarrassment, additional 
pressure and job stress, and risk to job security. It is important to note that intrinsic outcomes such as 
personal satisfaction from achieving the goals were as likely and as valuable to teachers as the extrinsic, 
formal elements of the programs (e.g. the bonus), and that teachers who rated intrinsic outcomes as highly 
desirable did not rate receiving a bonus as undesirable. Bonuses per se do not seem to be seen by these 
teachers as antithetical to the intrinsic motivators of the teaching job. Increased job stress, a potential 
consequence of trying to reach the goals, was also seen as about as unfavorable by teachers as the bonus 
was seen as favorable. 

While Kentucky teachers found the bonus per se desirable, they were not, on average, highly in 
favor of having the bonus program continue, nor was their self-reported motivation by the bonus high. 
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Teachers reporting higher levels of motivation from the bonus and a desire to see it continue tended to be 
those who were dissatisfied with their current salary, felt that use of bonuses and the bonus process was 
fair, and felt that teachers should decide how to divide up the bonus money. Those who reported a greater 
desire to withdraw from the bonus program were those who saw the program as less fair. 

While the desirability of receiving a bonus was rated relatively highly, teachers were less than 
certain that achieving the goals would lead to receiving the bonus. Teacher expectancy proved to be the 
primary predictor of subsequent school success, and yet teachers’ expectations that their efforts would 
result in goal achievement were weaker than expected. Teachers reported higher levels of individual and 
group expectancy in schools in which teachers perceived stronger assessment data feedback mechanisms, 
principal support or professional community, lack of goal conflict, and higher levels of perceived fairness 
of the program. Past performance in the program was important as well: teachers in schools with a 
history of success with the program reported higher levels of expectancy. 

These findings suggest that successful SBPA programs involve more than promising bonuses to 
teachers if performance goals are met. As the motivation model implies, making a group reward program 
work is a complex undertaking, requiring attention to all five elements of the motivation model for it to 
work (expectancy, instrumentality, valence, teacher competencies and enabling conditions). Because of 
this complexity, SBPA programs are not likely to be successful unless considerable attention is paid to 
their design, management and evaluation. The Kentucky and CMS programs were school-based, 
rewarded continuous improvement, included ongoing adjustments to address program fairness, and were 
part of more comprehensive reform strategies. However, they offered relatively small bonuses, suffered 
from political instability, and lacked sufficient attention to the creation of enabling conditions. It is 
important to note that the level of success of group reward programs varies in the private sector as well. 

It appears that a sizable minority of group incentive programs implemented in the private sector are 
judged unsuccessful (Abosch, 1998) or are discontinued (Bullock and Tubbs, 1990, Graham-Moore and 
Ross, 1990). The complexity of group reward programs is also illustrated by the fact that the private 
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sector literature has tended to be cautious in recormnending adoption when critical context conditions are 
not met (e.g. Wilson and Phalen, 1996, Orens and Elliott, 1997, Graham-Moore and Ross, 1990). 

Several other important policy implications for the design and implementation of SBPA programs 
emerge from our research, as well as private sector research on bonus systems. Here, we focus on the 
importance of teacher expectancy and enabling conditions, and some key issues for further exploration. 

Teacher confidence that effort will lead to goal achievement was a critical predictor of school 
success in our data. Yet expectancy was relatively low in both programs that we studied. Our data 
suggest that providing more positive organizational contexts that support goal achievement could enhance 
expectancy. An SBPA program should be one element of a comprehensive strategy to align resources 
and focus efforts on goal achievement. First, these efforts should include, but not be limited to providing 
meaningful and timely data to teachers on past performance, and training in how to interpret and utilize 
the information to improve teaching practice. 

Second, district and school leadership and policy reform efforts should consistently provide 
teachers with clear direction about what goals are valued, and a clear path toward goal achievement. 
Private sector literature has recognized the importance of the active commitment of site managers to the 
success of group reward programs (Bennett, 1998, Mastemak, 1997, Graham-Moore and Ross, 1990, 

Ross et al, 1992). Managers in successful programs seem to make the program central to their 
management efforts, and generate continuous interest and enthusiasm for the program. In both the 
Kentucky and CMS programs, this important role fell to principals, who may have had little or no 
guidance from the district or state as to how to carry it out. In our site interviews, there appeared to be a 
high level of variability in the extent to which the Kentucky and CMS principals “pushed” the program. 
While some were very proactive, others seemed genuinely at a loss as to how to rally their staff around 
the goals. 

Third, fairness was a key predictor of teacher expectancy, but average perceptions of program 
fairness were not high in either program we studied. As was the case in our data, private sector research 
is clear on the importance of perceptions of fairness on the motivational impact of reward programs 
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(Welboume and Gomez-Mejia, 1995, Cooper et al, 1992, Lawler, 1988, Hatcher et al, 1991, Gross and 
Barcher, 1993). It may be necessary to expend considerable effort explaining the rationale behind the 
performance measures, the level of the improvement goals, and the mechanics of the program to teachers 
to establish a basis for fairness perceptions. Reliability and “face validity” of assessments of student 
achievements and other measures of school performance are also likely to be crucial. Both the Kentucky 
and CMS programs included features to address fairness, such as setting goals based on growth from each 
individual school’s baseline scores, requiring improvements in high and low performing students, and in 
the case of CMS, attention to race equity issues. However, our research suggests that both programs 
needed to invest additional effort in enhancing perceptions of program fairness. 

Participation in program design and implementation by affected staff may also be a promising 
way to improve fairness perceptions and to ensure that improvement goals are not set too high as to seem 
unattainable, as well as to promote understanding and “buy-in” to the program. Again, private sector 
research (Bullock and Tubbs, 1990, White, 1979, Abosch, 1998) and prescription (O’Neill and Lough, 
1994, Wilson and Phalen, 1996) emphasize the importance of participation in design and implementation. 
Kellor and Odden (1999) describe a participative process used to design an SBPA program for the 
Cincinnati school district. 

We found a number of surprises in our data that suggest the need for further research and 
exploration. First, the connection between rewards and goal achievement was weak, and teacher 
instrumentality perceptions were low. Providing a consistent source of funding for, and a commitment to 
paying the bonus when it is earned might strengthen teacher perceptions of the achievement-bonus link. 
The recent experience in North Carolina is a good example of such a commitment. When that state’s 
SBPA program was implemented in 1997, about $80 million was budgeted for the awards. Because more 
schools than expected met their improvement targets, full funding required $120 million. Rather than 
scaling back the level of the awards, the legislature increased the budget to pay all teachers in improving 
schools the awards that they had been promised (Johnson et al, 1999). 
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Second, teachers indicated that the bonus was a weak motivator of performance. Additional 
research is needed to examine what effect a larger bonus might have. While the bonuses used in 
Kentucky and CMS were close to the lower end of the 3-5% of base pay range common to private sector 
programs (McAdams and Hawk, 1994) our interviews with CMS teachers suggest that on average this 
was not perceived as a significant amount. To reach a meaningful threshold for most teachers, larger 
bonuses may be needed. However, some reports (Hall and Caffarella, 1997; Kelley, 1996; Kelldr and 
Odden, 1998) suggest that teachers in some situations do see smaller bonuses as significant. The 
difference may be that in these cases, differences in program design may have reduced negative outcomes 
associated with program participation, improving the overall motivational power of the program. 
Additional research is needed to explore the effects of larger bonuses on teacher motivation and student 
achievement. 

Third, the proxies for knowledge and skills did not prove to be significant predictors of 
expectancy in our models. Nonetheless, many reformers have suggested that teachers lack the skills they 
need to teach to high standards (e.g. Con, 1996, National Commission on Teaching and America’s Future, 
1996). Thus, additional professional development and knowledge and skill-based pay incentives provide 
promising avenues to encourage teachers to develop knowledge and skills and thereby enhance teacher 
expectancy to improve student performance. Additional research and experimentation is needed to 
examine how knowledge and skill-based pay plans may be linked to SBPA programs to reward teachers 
both for the development of their own knowledge and skills as well as for outcomes produced. 
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Table 1 . Means and Standard Deviations of Responses on Goal-Related Scales 





CMS 

Mean 


CMS 

Std. Deviation 


Kentucky 

Mean 


Kentucky 
Std. Deviation 


Goal Cominitment 


3.9 


0.8 


3.9 


0.7 


Goal Clarity 


3.4 


1.0 


3.6 


0.8 


Goal Conflict 


2.8 


I.O 


3.2 


1.1 



Note: Ratings are on a scale of 1 to 5. 
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Table 2. Teacher Performance Goal Targets in CMS and Kentucky 



CMS 

During this year, did you: 


All 

Respondents 


Respondents in 
Schools Rewarded 
in Prior Year 


Respondents in 
Schools Not 
Rewarded in 
Prior Year 


not try to achieve goals 


5.6 


3.5 


7.5 


try to avoid decline in student performance measures 


7.2 


7.5 


7.0 


try to achieve “Outstanding'* status (partial reward ) 


13.4 


9.3 


17.1 


try to achieve “Exemplary ” status (full reward) 


73.8 


79.6 


68.4 








Respondents in 


Kentucky 




Respondents in 


Cycle 2 


During this year, did you: 


All 


Cycle 2 


Nonreward 




Respondents 


Reward Schools 


Schools 


not try to achieve goals 


1.5 


2.2 


1.2 


try to avoid decline in KIRIS scores 


10.0 


7.3 


11.6 


try to achieve “Improving" Status 


10.8 


6.3 


13.6 


try to achieve “Successful" Status 


24.4 


12.0 


32.2 


try to achieve “Reward” Status 


53.4 


72.2 


41.4 



ERIC 
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Table 3. Percentages of Teachers Indicating Goals Were Becoming Harder to Achieve 





Reward 
Prior Cycle 


Nonreward 
Prior Cycle 


Percent of CMS respondents indicating that: 


this year’s goals are much or somewhat harder 


67.4 


43.5 


next year’s goals will be much or somewhat harder 


59.5 


44.1 


Percent of Kentucky resnondents strongly agreeing or agreeing that: 


this cycle’s goals will be harder to achieve 


73.5 


63.5 


next cycle’s goals will be even harder to reach 


70.9 


65.7 
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Table 4. Desirability of Outcomes Associated with School-Based Performance Award Programs 



Outcome 


CMS Mean 


CMS Std. 
Deviation 


Kentucky 

Mean 


Kentucky Std. 
Deviation 


Goal Attainment Rewards 


Personal satisfaction from improved student 
performance 


4.6 


0.6 


4.4 


0.6 


Receiving a bonus (a) 


4.5 


0.7 


4.1 


1.1 


Receiving funds for school improvement 


4.4 


0.7 


4.1 


0.9 


Personal satisfaction of meeting goals 


4.3 


0.8 


4.2 


0.8 


Public recognition for meeting goals 


4.1 


0.9 


4.0 


0.9 


Learning 


Having your students learn new skills 


4.5 


0.6 


4.4 


0.7 


Having clear school- wide goals 


4.4 


0.7 


4.3 


0.7 


Working cooperatively with other teachers on 
curriculum and instruction 


4.3 


0.7 


4.0 


0.8 


Having additional opportunities for professional 
development 


4.3 


0.7 


3.9 


0.8 


Sanctions 


Public criticism or embarrassment due to not 
achieving goals (b) 


4.4 


1.2 


1.6 


0.8 


Risk to job security (b) 


3.9 


1.1 


1.6 


0.8 


Intervention (b)(c) 


3.7 


1.3 


1.8 


1.0 


Loss of professional pride due to not achieving goals 
(b) 


3.6 


1.2 


1.8 


0.8 


Stress 


Putting in more hours 


2.0 


0.9 


1.9 


0.9 


Less freedom to teach things unrelated to goals 


2.0 


0.9 


1.9 


0.9 


More pressure and job stress 


1.6 


0.8 


1.5 


0.8 



(a) $1,000 in CMS, $2,000 in Kentucky 

(b) CMS respondents were asked to rate the desirability of avoiding these outcomes, while Kentucky respondents were asked to 
rate the desirability or undesirability of the outcomes themselves. 

(c) CMS respondents were asked about having their school designated a Priority School, while Kentucky respondents were asked 
about having a Distinguished Educator assigned to their school. 



Source: Heneman & Milano wski, 1999 





Table 5. Average Perceived Probability of Outcomes Associated with SBPA Programs 

Outcome Probability if School Probability if You Try to Probability if School 

Meets Goals Meet Goals Does Not Meet Goals 





CMS 


KY 


CMS 


KY 


Goal Attainment Rewards 
Receiving a bonus (a) 


72.8 


54.8 






Personal satisfaction of meeting goals 


79.4 


73.9 






Personal satisfaction from improved student 


78.9 


73.6 






performance 

Public recognition for meeting goals 


67.6 


62.1 






Receiving funds for school improvements 


45.2 


49.6 






Le amine 

Working cooperatively with other teachers on 
curriculum and instruction 






72.9 


71.5 


Feel you are working toward clear school-wide 
goals 






73.6 


63.3 


Having additional opportunities for professional 
development 






63.5 


68.5 


Having your students learn new skills 






54.6 


59.9 


Stress 

More pressure and job stress 






73.6 


84.4 


Putting in more hours 






65.8 


81.9 


Less freedom to teach things unrelated to goals 






68.9 


74.9 



Sanctions 

Public criticism or embarrassment due to not 


53.8 


68.6 


achieving goals 
Intervention 


46.5 


75.9 


Loss of professional pride due to not achieving 
goals 


44.9 


61.0 


Risk to job security 


31.8 


50.2 



(a) $1,000 in CMS, $2,000 in Kentucky. 



Source: Heneman & Milanowski, 1999 





Table 6. Teachers’ Perceptions about the Fairness of the SBPA Programs 



Item 


CMS 

Mean“ 


CMS 

Std. 

Deviation 


Kentucky 

Mean’* 


Kentucky 

Std. 

Deviation 


Fair to hold teachers accountable for student achievement. 


3.3 


1.1 


2.5 


1.2 


Fair for teachers who increase student achievement to get a 
bonus. 


3.6 


1.1 


2.8 


1.2 


School has a fair chance to win award (2 item scale) 


2.8 


1.2 


2.9 


1.1 


Program is procedurally fair (3 item scale) 


2.6 


0.9 


2.4 


0.9 



(a) Response scale was l=Strongly Disagree... 5=Strongly Agree. 
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Table 7. Average Individual and Group Expectancy: Kentucky and CMS Teachers 





Kentucky 
Average * 


Kentucky 
Std. Deviation 


CMS 

Average* 


CMS 

Std. Deviation 


Individual Expectancy - Reward 


39.2 


26.8 


55.6 


29.5 


Group Expectancy - Reward 


52.7 


27.3 


62.0 


29.1 


Individual Expectancy- Avoid Decline 


44.5 


27.1 


- 




Group Expectancy - Avoid Decline 


60.9 


27.3 


- 


- 



* Response scale ranged from 0 (Highly Unlikely) to 100 (Highly Likely) in 10 point steps. 



Table 8. Regression Estimates (Standardized Coefficients) of Factors Affecting Teacher 



Individual Expectancy Perceptions in CMS and Kentucky 





Teacher Expectancy 
(Charlotte-Mecklenburg) 


Teacher Expectancy 
(Kentucky) 

Model 1 Model 2 


School Characteristics 


High School 


-.235*** 


-.074* 


-.072* 


Middle School 


-.083** 


-.036 


-.029 


School Size 


.060 


-.004 


-.013 


Free Lunch 


-.023 


-.028 


-.040 


Reward History 
Organizational Context 


.031 


.148*** 


.137*** 


Feedback 


.175*** 


.173*** 


.179*** 


Alignment 


-.016 


-.016 


-.023 


Community (a) 
Community (b) 


.021 


.112*** 


.121*** 


Participation 


-.002 


.043 


.028 


Principal Support 


.110** 


.010 


.034 


Goal Conflict 


-.051 


-.102*** 


-.088*** 


Teacher Attitudes 


Fairness 


.269*** 


.205*** 


.233*** 


Goal Clarity 


.048 


-.038 


-.053 


Teacher Characteristics 


Experience 


-.020 


.008 


.009 


Education 


-.047 


-.049 


-.042 


N 


900 


1393 


1384 


Model R-squared 


.237 


.291 


.296 



*** Statistically Significant at the .01 Level (a) Same question as in CMS 

** Statistically Significant at the .05 Level (b) Augmented scale with additional questions 

* Statistically Significant at the .10 Level 

Source: Kelley, 1999 
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Table 9. The Effect of Teacher Expectancy and Expected Outcome Value 
(Instrumentality times Valence) on School Performance 



CMS (n=l 14 schools) 


Controls 

Only 


Value of 
Bonus 


Goal 

Attainment 


Learning 


Sanctions 


Stress 


All 4 Types 
of 

Outcomes 


Log Enrollment 


..337*** 


..275*** 


..267*** 


• 219 *** 


-.291*** 


-.276*** 


-.307*** 


Percent Free/Reduced Lunch 


..528*** 


..522*** 


-.514*** 


-.481 **♦ 


-.508*** 


-.533*** 


..484*** 


Expectancy 




.265*** 


.318*** 


.374*** 


.295*** 


.301*** 


.369*** 


Expected Value of Bonus 




.051 












Goal Attainment Outcomes 






-.053 








.049 


Learning Outcomes 








-.177* 






-.183 


Sanctions 










.134 




.132 


Stress Outcomes 












-.145* 


-.158* 


R- Squared 


.199 


.281 


.281 


.299 


.295 


.299 


.338 



Kentucky (n=204 schools) 


Controls 

Only 


Value of 
Bonus 


Goal 

Attainment 


Learning 


Sanctions 


Stress 


All 4 
Types of 
Outcomes 


Log Enrollment 


-.143** 


-.129** 


-.142** 


-.145** 


-.134*** 


-.145** 


-.129* 


Percent Free/Reduced Lunch 


-.484*** 


-.496*** 


-.512*** 


-.523*** 


-.517*** 


-.519*** 


-.514*** 


Cycle 2 KIRIS Index 


-.601*** 


-.710*** 


-.732*** 


-.740*** 


-.680*** 


-.728*** 


-.727*** 


Expectancy 




.290*** 


.283*** 


.251*** 


.222*** 


.266*** 


.298*** 


Expected Value of Bonus 




-.137** 












Goal Attainment Outcomes 






-.087 








-.140 


Learning Outcomes 








-.015 






.098 


Sanctions 










-.068 




-.082 


Stress Outcomes 












-.088 


-.040 


R- Squared 


.283 


.342 


.333 


.328 


.326 


.334 


.345 



*** Statistically Significant at the .01 Level 
** Statistically Significant at the .05 Level 
* Statistically Significant at the . 10 Level 



best copy available 
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APPENDIX A 



Table Al. Items Combined to Produce Goal Commitment, Goal Clarity, Goal Focus, and Goal Conflict 
Scales and the Internal Consistency Reliability of the Scales 



Scale and Items 



Coefficient 

Alpha 



Goal Commitment 

CMS .73 

It’s hard to take the Benchmark Goals seriously, (scale reversed) 

I am strongly committed to pursuing my school’s Benchmark Goals. 

Quite frankly, I don’t care if my school achieves its Benchmark Goals or not. 

(scale reversed) 

Kentucky .74 

It’s hard to take the accountability goals seriously, (scale reversed) 

I am strongly committed to pursuing my school’s accountability goals. 

Quite frankly, I don’t care if my school achieves its accountability goals or not. (scale reversed) 

Goal Clarity 

CMS .90 

I have a clear understanding of what my school’s Benchmark Goals are. 

I know how much each aspect of student achievement needs to improve in order to reach the 
Benchmark Goals for this year. 

1 have a good understanding of how student achievement related to the Benchmark Goals is 
measured in the subject(s) I teach. 

I could explain my school’s Benchmark Goals to others if asked. 

Kentucky .85 

I have a clear understanding of what my school’s accountability goals are. 

I know how much each aspect of student achievement needs to improve in order to reach the 
accountability goals for this cycle. 

I have a good understanding of how student achievement is measured by KIRIS in the subject(s) I 
teach. 

I could explain my school’s accountability goals to others if asked. 

Goal Focus .69 

Kentucky 

The accountability goals provide a focus for my teaching efforts. 

The accountability goals tell us what is most important for the school to accomplish. 

Goal Conflict 

CMS .75 

This year’s Benchmark Goals conflict with other important goals of our school. 

Working toward our school’s Benchmark Goals prevents me from working toward other 
important educational goals. 

Kentucky .80 

The accountability goals conflict with other important goals of our school. 

Working toward our school’s accountability goals prevents me from working toward other 

important educational goals. 
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Table A2. Items Combined to Produce Fairness of SBPA Programs Scales 
and the Internal Consistency Reliability of the Scales 





CMS 


Kentucky 


Scale and Items 


Coefficient 


Coefficient 




Alpha 


Alpha 


School has a fair chance to win award (2 item scale) 


.78 


.71 


Our school has as much chance of being a reward school as any other. 

Our school has less chance of achieving its accountability goals than others because 
of our student population (scale reversed). 






Program is procedurally fair 

As a way to measure school performance, the accountability goals are fair. 
The way our school’s accountability goals were established is fair. 

Overall, the way the accountability program is run is fair. 


.85 


.84 




44 



46 



Table A3. Items Used in Scales for Analysis of Teacher Expectancy 



Dependent Variables: 

Individual Teacher Expectancy 

CMS: If you were to make the highest level of effort you would be comfortable with (that you could sustain for the 
whole year), what is the probability or likelihood of you school meeting its Benchmark Goals? (0%, 10%, 20%,. . . 
100 %) 

Kentucky: What is the chance, probability, or likelihood that if you personally were to make the highest level of 
effort you could, your school would exceed its accountability goals by enough to be a “Reward” school? (0%, 10%, 
20 %,. . . 100 %) 

Teacher Group Expectancy: 

CMS: If you and your colleagues were to work together as hard as you could, what is the probability or likelihood of 
you school meeting its Benchmark Goals? (0%, 10%, 20%, . . . 100%) 

Kentucky: What is the chance, probability, or likelihood that if you and your colleagues were to work together as 
hard as you could, your school would exceed its accountability g oals by enough to be a “Reward” school? (0%, 

10 %, 20 %, . . . 100 %) 

Independent Variables: 

School Characteristics: 

School Size : Student Headcount Enrollment 
Free Lunch : Percent Free and Reduced Lunch 

Reward History : for CMS, 1995-96 performance on the Benchmark Goals (Reward=l, Nonreward=0); for 
Kentucky: Cycle 2 performance on the Kentucky School-Based Accountability Program (Reward=l, 
Nonreward=0) 

Enabling Conditions: Scales created from responses to the following five-point likert scale questions: 

Feedback 

CMS: Concerning the feedback you receive about the Benchmark Goals Program, to what extent: 

- Do you receive the results of the tests or other assessments that are used to measure whether your school achieved 
its Benchmark Goals in the subjects you teach? 

- Do you know how to interpret the results of the assessments related to the Benchmark Goals? 

- Are the results of these assessments useful to you in figuring out how to help achieve your school’s Benchmark 
Goals? 

Kentucky: Concerning the feedback you receive about the accountability goals, to what extent: 

- Do you receive KIRIS results relevant to the subjects you teach? 

- Do you know how to interpret the results of the KIRIS assessments in relation to the accountability goals? 

- Are the results of the KIRIS assessments useful to you in figuring out how to help achieve your school’s 
accountability? 

Alignment 

At your school, to what extent: 

CMS 

- Are teachers evaluated based on their efforts to achieve the school’s Benchmark Goals? 

- Are new curriculum development efforts linked to the Benchmark Goals assessment? 

- Are teachers provided time to plan together about how to achieve the Benchmark Goals? 
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- Has your school mobilized its resources to meet its Benchmark Goals? 

- Do professional development activities in your school focus on helping you acquire knowledge and skills related to 
meeting the Benchmark Goals? 

Kentucky 

- Are teachers evaluated based on their efforts to achieve the school’s accountability goals? 

- Are new curriculum development efforts linked to KIRIS? 

“ Are teachers provided time to plan together about how to achieve the accountability goals? 

- Has your school mobilized its resources to meet its accountability goals? 

- Do professional development activities in your school focus on helping you acquire knowledge and skills related to 
meeting the accountability goals? 

Community (a) : At your school, to what extent do teachers in your school work together to improve teaching 
knowledge and skills? 

Community (b) (for Kentucky only): 

At your school, to what extent do teachers in your school work together to improve teaching knowledge and skills? 
Do teachers seek the advice of other teachers when they have a teaching problem? 

Do teachers discuss ideas about how to improve teaching and student learning among themselves? 

Do teachers share a vision of student learning and what it takes to accomplish it? 

Do teachers hold one another accountable to work hard to increase student learning? 

Teacher Participation (both sites): At your school, to what extent: 

- Do teachers make important decisions about curriculum and instruction? 

- Do teachers make important decisions about the allocation of the school’s budget? 

- Do teachers make important decisions about the hiring of new teachers? 

- Are teachers provided with the information they need to participate meaningfully in school-level management 
decisions? 

Principal Support 

To what extent is your school’s principal: 

CMS 

- committed to achieving your school’s Benchmark Goals? 

- supportive of your efforts to achieve the school’s Benchmark Goals? 

- willing to work with teachers to help them understand how to achieve the Benchmark Goals? 

Kentucky 

- committed to achieving your school’s accountability goals? 

- supportive of your efforts to achieve the school’s accountability goals? 

- willing to work with teachers to help them understand how to achieve the accountability goals? 

Goal Conflict (extent of agreement or disagreement with the following statements) 

CMS: 

- This year’s Benchmark Goals conflict with other important goals of our school. 

- Working toward our school’s Benchmark Goals prevents me from working toward other important educational 
goals. 

Kentucky: 

- The accountability goals conflict with other important goals of our school. 

- Working toward our school’s accountability goals prevents me from working toward other important educational 
goals. 
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Teacher Attitudes: Scale created from responses to the following five-point likert scale questions: 

Fairness 

(extent of agreement or disagreement with the following statements): 

CMS 

- The way our school’s Benchmark Goals were established was fair. 

- As a way to measure school performance, the Benchmark Goals are fair. 

- Overall, the way the Benchmark Goals program is run is fair. 

Kentucky 

- The way our school’s accountability goal was established was fair. 

- As a way to measure school performance, the accountability goals are fair. 

- Overall, the way the accountability program is run is fair. 

Goal Clarity 

(extent of agreement or disagreement with the following statements): 

CMS 

- 1 have a clear understanding of what my school’s Benchmark Goals are. 

- 1 know how much each aspect of student achievement needs to improve in order to reach the Benchmark Goals for 
this year. 

- 1 have a good understanding of how student achievement related to the Benchmark Goals is measured in the 
subject(s) I teach. 

- 1 could explain my school’s Benchmark Goals to others if asked. 

Kentucky 

- 1 have a clear understanding of what my school’s accountability goals are. 

- 1 know how much each aspect of student achievement needs to improve in order to reach the accountability goals 
for this cycle. 

- 1 have a good understanding of how student achievement is measured by KIRIS in the subject(s) I teach. 

- 1 could explain my school’s accountability goals to others if asked. 

Teacher Characteristics 

Experience : Total number of years of teaching experience. 

Education : 1= Highest degree obtained is MA/MS, MA/MS plus credits, or EdD or PhD, else = 0 
Source: Kelley, 1999 
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